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Detailed Action 
Preliminary Amendment 

1 . The preliminary amendment filed on 7/1 0/2003 has been entered . 

Title 

2. The title of the invention is not descriptive. A new title is required that is clearly 
indicative of the invention to which the claims are directed. 

Drawings 

3. The drawings are objected to because: 

In figure 5 step S5050 "5010" should be "2010" since paragraph 0045 describes 
virtual object 2010 not virtual object 5010 and since 5010 in figure 5 is a step and this 
step is not used in the distance determination of step 5050; 

In figure 7 step S70 10 describes 4 or more and paragraph 0064 describes 4 or 
more. It is not clear from figure 7 how step S7010 can determine more than 4 since it 
can only determine if 4 is present and then exists to step S7020; 

In figure 1 1 step Sb010 describes 8 and paragraph 0080 describes 2 or more. It 
is not clear from figure 1 1 how step Sb010 can determine more than 2 since it can only 
determine if 2 is present and then exists to step Sb020; 

In figure 13 2 elements are designated by the same reference number 2010'. It 
is possible applicant intended for 2010' to be d010'; and 

In figure 14 eO has no corresponding description in the Description of the 
Preferred Embodiments; 
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Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in 
reply to the Office action to avoid abandonment of the application. Any amended 
replacement drawing sheet should include all of the figures appearing on the immediate 
prior version of the sheet, even if only one figure is being amended. The figure or figure 
number of an amended drawing should not be labeled as "amended." If a drawing figure 
is to be canceled, the appropriate figure must be removed from the replacement sheet, 
and where necessary, the remaining figures must be renumbered and appropriate 
changes made to the brief description of the several views of the drawings for 
consistency. Additional replacement sheets may be necessary to show the renumbering 
of the remaining figures. The replacement sheet(s) should be labeled "Replacement 
Sheet" in the page header (as per 37 CFR 1.84(c)) so as not to obstruct any portion of 
the drawing figures. If the changes are not accepted by the examiner, the applicant will 
be notified and informed of any required corrective action in the next Office action. The 
objection to the drawings will not be held in abeyance. 

Claim Rejections - 35 USC §112 

4. The following is a quotation of the second paragraph of 35 U.S.C. 1 12: 

The specification shall conclude with one or more claims particularly pointing out and distinctly 
claiming the subject matter which the applicant regards as his invention. 

5. Claims 3, 9, 15, and 18 are rejected under 35 U.S.C. 112, second paragraph, as 
being indefinite for failing to particularly point out and distinctly claim the subject matter 
which applicant regards as the invention. 
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Claim 3 is indefinite because the does not clearly state the process that is 
occurring when applicant claims "translation transformation following the constraining 
shape". Similarly the process occurring when applicant claims "rotation transformation 
on an axis" since it does not claim which claimed element is rotated. 

Claim 9 is indefinite because "the computer program" lacks antecedent basis in 
the claim. 

Claim 15 is indefinite because "the computer program" lacks antecedent basis in 
the claim. 

Claim 18 is very similar to claim 3 and this claim is indefinite for the reasons 
given for claim 3. 

Claim Rejections - 35 USC § 102 

6. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that 
form the basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(b) the invention was patented or described in a printed publication in this or a foreign country or in public 
use or on sale in this country, more than one year prior to the date of application for patent in the United 
States. 

7. Claims 1-19 are rejected under 35 U.S.C. 102(b) as being anticipated by 
Yoshifumi Kitamura and Fumio Kishino, Consolidated Manipulation of Virtual and Real 
Objects, September 1997, Proceedings of the ACM symposium on Virtual reality 
software and technology, pages 133-138. Kitamura teaches an augmented reality 
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system that uses object constraints to control the visual interaction between the virtual 

objects and the real objects. 

A detailed analysis of the claims follows. 

Claim 1: 

Kitamura teaches an information processing device for aiding control operations relating 
to position and orientation of a virtual object positioned in three-dimensional space, said 
device comprising: 

image-taking means for taking images in real space in order to display the_said 
virtual object in a superimposed manner on real space (The three paragraphs 
found in section 2 on page 134 of Kitamura teaches image taking 
means: measuring the real objects, using a range image/ using 
stereo images, or graphical images generated from the geometric 
shapes, ); 

synthesizing means for synthesizing a virtual object with the said taken image 

(The introduction on page 133 second full paragraph discusses 
augmented reality which synthesizes a virtual object with a taken 
real object.)] 

operating means for controlling the position and orientation of the said virtual 
Object (The first paragraph in section 5 on page 135 describes the 
user using a 6 DOF tracker device to manipulate the virtual 
objects .); and 
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aiding means for obtaining a three-dimensional position of real space from 
external instructions (For example Kitamura discusses in section 2 in 
the first paragraph using conventional modeling software after 
precisely measuring the size or length of the real object by 
hand. Measuring by hand generates external instructions . 
Another source of external instructions is the range image or 
stereo image obtained by multiple cameras described also in 
section 2 first paragraph.), and obtaining a constraining shape for aiding in 
control operations for the position and orientation of the said virtual object (The shape 
generated from the external instructions constraints the 
interaction of the virtual object with the real object, see 
sections 5.1 to 5.4.); 

wherein the position and orientation of the said virtual object are controlled, by 
instructions from said operating means, based on constraint conditions based on the 
constraining shape obtained by said aiding means (The shape of the real 
objects are used to constrain the movement of the virtual object 
by giving the real object a shape that the virtual object 
interacts with in a constrained manner. Sections 5.1 to 5.4 
discusses manipulation of the virtual object based upon 
constraint conditions based on the shape of the real object.). 
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Claim 2: 

Kitamura teaches an information processing device according to Claim 1, wherein the 
constraining shape is a shape in which a three-dimensional 

position in real space (Section 2 first paragraph discusses 3D shapes .) obtained 
by external instructions (Section 2 first paragraph discusses determining the 
3D shapes by hand or by multiple cameras.) is configured of an apex (The phrase 
an apex is singular and is met at least by taking in points of the 
real object by hand .) or a component plane (On page 136 in the text above 
figure 2 determining a plane and using the plane to constrain movement of 
the virtual object is discussed with regards to figure 2.). 

Claim 3: 

Kitamura teaches an information processing device according to Claim 1 , wherein the following 
control operations of the position and orientation of the said virtual object can be made using said 
operating means: 

translation transformation following the constraining shape (See figures 4(a) and 
5.); and 

rotation transformation on an axis which is a normal vector at a plane where the 
constraining shape and the virtual object come into contact (After the virtual object is 
snapped on to the face of the real object the virtual objects motion 
is constrained to translation on the real object *s surface and to 
rotation on the real object's surface, see the first full paragraph on 
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page 135 in section 5.1 and the first full paragraph on page 137 above 
section 5.3.2.). 

Claim 4: 

Kitamura teaches an information processing device according to Claim 1 , wherein the virtual 
object is subjected to three-dimensional transformation by control operations of said operating 
means (The 6DOF manipulator controls the movement of the virtual object 
in three dimensions . ), such that the position of said operating means on a two-dimensional 
screen (Kitamura % s article was published in 1991, in the year 1997 most 
display were two dimensional, thus, Kitamura' s augmented reality is 
displayed on a two dimensional screen.) and the position of the said virtual object on 
a two-dimensional screen are synchronized (The 6DOF manipulator is synchronized 
with the virtual object since it controls the position of the virtual 
object on the two dimensional screen.). 

Claim 5: 

Kitamura teaches an information processing method for aiding control operations relating to 
position and orientation of a virtual object positioned in three-dimensional space, said method 
comprising: 

a measurement step obtaining external parameters indicating the position and orientation of 
image-taking means in order to fixedly display the virtual object in real space ( The three 
paragraphs found in section 2 on page 134 of Kitamura teaches 
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image taking means: measuring the real objects, using a range 
image, using stereo images , or graphical images generated from 
the geometric shapes. Measuring the real objects determines the 
position and orientation of image -taking means since the position of 
the hand held unit's position and orientation are used to calculate 
the position of the real objects in the augmented reality scene. The 
range image and stereo images inherently provide position and 
orientation of the multiple cameras in order to calculate the position 
of the real objects in the augmented reality scene. These external 
parameters determines the position of the virtual object at least due 
the shape based constraints generated from these external parameters.)] 

an operating step for controlling the position and orientation of the virtual object (The 
first paragraph in section 5 on page 135 describes the user using 
a 6 DOF tracker device to manipulate the virtual objects .)] 

an aiding step for adding constraints to the control operations of an operating means so as 
to aid the control operations on the virtual object (The shape generated from the 
external instructions constraints the interaction of the virtual 
object with the real object, see sections 5.1 
to 5.4.)\ 

an input step for inputting a constraining shape to be a reference for generating the 
constraints (The shape of the real objects are input as discussed in 
section 2. The input shapes are used to constrain the movement 
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of the virtual object. Sections 5.1 to 5.4 discusses manipulation 
of the virtual object based upon constraint conditions based on 
the shape of the real object.): 

a synthesizing step for synthesizing pictures of real space obtained by image-taking means, 
and pictures of the said virtual object estimated from the position and orientation of the said image- 
taking means (The introduction on page 133 second full paragraph 
discusses augmented reality which synthesizes a virtua.1 object with 
a taken real obj ect.)\ 

wherein the position and orientation of the said virtual object in said operating step is 
transformed in said aiding step with the constraining shape as a reference, thereby aiding control 
operations (The shape of the real objects are used to constrain the 
movement of the virtual object by giving the real object a shape 
that the virtual object interacts with in a constrained manner. 
Sections 5.1 to 5.4 discusses manipulation of the virtual object 
based upon constraint conditions based on the shape of the real 
object.). 

Claim 6: 

Kitamura teaches an information processing method according to Claim 5, wherein said input 
step obtains three-dimensional position in real space from external instructions (For exampl e 
Kitamura discusses in section 2 in the first paragraph using 
conventional modeling software after precisely measuring the size 
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or length of the real object by hand. Measuring by hand 
generates external instructions . Another source of external 
instructions is the range image or stereo image obtained by 
multiple cameras described also in section 2 first paragraph.). 

Claim 7: 

Kitamura teaches an information processing method according to Claim 6, wherein a virtual 
image indicating the input virtual shape is synthesized with an image of the said real space and 
pictures of the said virtual object (See the discussion of claim 6. Also refer to 
figure 2, 4(a), 5, and 6(a).). 

Claim 8: 

Kitamura teaches a computer program, wherein the information processing method 
according to Claim 5 is executed by a computer device (This article is directed 
to computers that generate the augmented reality scene since it 
was published by ACM for a symposium on virtual reality software 
and technology and since at page 133 in the last sentence in the 
second paragraph of section 1 software/hardware is discussed.). 

Claim 9: 

Kitamura teaches a computer-readable recording medium, storing the computer 
program according to Claim 8 (Software causing a computer to perform 
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Kitamura' s augmented reality is inherently stored in a computer 
readable recording medium,). 

Claim 10: 

Kitamura teaches an information processing method for controlling the position and 
orientation of a virtual object in compounded real space obtained by synthesizing 
pictures of real space and a virtual object, said method comprising: 

a step for obtaining a constraining shape serving as constraint conditions, from a 
plurality of positions in real space instructed by a user using an operating unit capable of 
obtaining three-dimensional information (The three paragraphs found in 
section 2 on page 134 of Kitamura teaches image taking means for 
obtaining 3D shape of real objects: measuring the real objects, 
using a range image, using stereo images, or graphical images 
generated from the geometric shapes. Kitamura discusses in 
section 2 in the first paragraph using conventional modeling 
software after precisely measuring the size or length of the real 
object by hand. Measuring by hand generates external 
instructions . Another source of external instructions is the 
range image or stereo image obtained by multiple cameras 
described also in section 2 first paragraph. The shape of the 
real objects are used to constrain the movement of the virtual 
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object by giving the real object a shape that the virtual object 
interacts with in a constrained wanner.)] 

a step for changing the position and orientation of the said virtual object according 
to instructions from the user, with the obtained constraining shape as constraint 
conditions (The first paragraph in section 5 on page 135 describes 
the user using a 6 DOF tracker device to manipulate the virtual 
objects. The shape of the real objects are used to constrain the 
movement of the virtual object by giving the real object a shape 
that the virtual object interacts with in a constrained manner. 
Sections 5.1 to 5.4 discusses manipulation of the virtual object 
based upon constraint conditions based on the shape of the real 
object. The shape generated from the external instructions 
constraints the interaction of the virtual object with the real 
object, see sections 5.1 to 5.4.)] and 

a step for synthesizing an image of virtual object generated according to the 
changed position and orientation, with pictures of real space (The introduction on 
page 133 second full paragraph discusses augmented rea.lity which 
synthesizes a virtual object with a taken real object. Section 5.1 
discusses after movement is detected by the 6DOF manipulator the 
virtual object is moved according to the constraints.). 
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Claim 11: 

Kitamura teaches an information processing method according to Claim 10, wherein a 
virtual image indicating the input virtual shape is synthesized with the pictures of real 
space (See the discussion of claim 10. Also refer to figure 2, 4(a), 
5, and 6 (a) .). 

Claim 12: 

Kitamura teaches an information processing method according to Claim 10, wherein the 
constraining shape is a plane (On page 136 in the text above figure 2 determining 
a plane and using the plane to constrain movement of the virtual object 
is discussed with regards to figure 2.). 

Claim 13: 

Kitamura teaches an information processing method according to Claim 10, wherein change of 
the position and orientation of the virtual object is carried out by changing the position and 
Orientation Of the operating unit (The 6 DOF tracker is an operating unit. The 
user using the 6 DOF tracker device manipulates the virtual 
objects .). 

Claim 14: 

Kitamura teaches a computer program, wherein the information processing method according to 
Claim 10 is executed by a computer device (This article is directed to computers 
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that generate the augmented reality scene since it was published 
by ACM for a symposium on virtual reality software and technology 
and since at page 133 in the last sentence in the second 
paragraph of section 1 software/hardware is discussed.). 

Claim 15: 

Kitamura teaches a computer-readable recording medium, storing the computer program 
according to Claim 14 (Software causing a computer to perform Kitamura' s 

r- 

augmented reality is inherently stored in a computer readable 
recording medium.). 

Claims 16-19: 

Claims 1 6-19 are device claims that correspond to device claims 1-4 with the difference being that 
device claim \4 included means plus function language and device claims 16-19 lack the mean 
plus function terminology. Device claims 1 6-1 9 replaced the word means with the word unit. 

Claim 16: 

Kitamura teaches an information processing device for aiding control operations relating to 
position and orientation of a virtual object positioned in three-dimensional space, said device 
comprising: 

an image-taking unit for taking images in real space in order to display the virtual object in a 
superimposed manner on real space (The three paragraphs found in section 2 
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on page 134 of Kitamura teaches image taking unit: measuring the 
real objects, using a range image, using stereo images, or 
graphical images generated from the geometric shapes. ); 

a synthesizing unit for synthesizing a virtual object with the taken image ( The 
introduction on page 133 second full paragraph discusses augmented 
reality which synthesizes a virtual object with a taken real object 
in a unit of the augmented reality system.)] 

an operating unit for controlling the position and orientation of the virtual object (The 
first paragraph in section 5 on page 135 describes the user using 
a 6 DOF tracker device to manipulate the virtual objects. The 6 
DOF tracker and associated means is a unit of the augmented reality 
system.)] and 

an input unit for obtaining a three-dimensional position of real space from external 
instructions (For example Kitamura discusses in section 2 in the first 
paragraph using conventional modeling software after precisely 
measuring the size or length of the real object by hand. 
Measuring by hand generates external instructions . Another 
source of external instructions is the range image or stereo 
image obtained by multiple cameras described also in section 2 
first paragraph. The means for obtaining the position is a unit 
of the augmented reality system .), and obtaining a constraining shape for aiding in 
controlling the position and orientation of the virtual object (The shape generated from 
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the external instructions constraints the interaction of the 
virtual object with the real object, see sections 5.1 to 5.4. 
This means for obtaining the position and constraining- shape is a 
unit of the augmented reality system.)] 

wherein the position and orientation of the virtual object are controlled by instructions from 
said operating unit, based on constraint conditions based on the constraining shape obtained by 
said input unit (The shape of the real objects are used to constrain the 
movement of the virtual object by giving the real object a shape 
that the virtual object interacts with in a constrained manner. 
Sections 5.1 to 5.4 discusses manipulation of the virtual object 
based upon constraint conditions based on the shape of the real 
object.). 

Claim 17: 

This claim is identical to claim 2 and it is rejected for the reasons given for claim 2. See the 
discussion of claim 2. 

Claim 18: 

This claim is substantially identical to claim 3 with the exception this claim claims "operating unit 1 . 
As stated for claim 1 6 the operating unit is the 6 DOF tracker and associated means. Thus, claim 
1 8 is rejected for the reasons given for claim 3. See the discussion of claim 3. 
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Claim 19: 

This claim is substantially identical to claim 4 with the exception this claim claims "operating unit". 
As stated for claim 1 6 the operating unit is the 6 DOF tracker and associated means. Thus, claim 
1 9 is rejected for the reasons given for claim 4. See the discussion of claim 4. 



Prior Art 

8. The prior art made of record and not relied upon is considered pertinent to 
applicant's disclosure. 

Sauer et al., U.S. Patent Application Publication No. 2002/0140709, teaches an 
augmented reality system that constrains the movement of the virtual object by placing 
virtual guides in the augmented scene. 

Bimber, U.S. Patent No. 6,803,928, teaches an augmented reality system that 
simulates the interaction between real and virtual objects, column 22 lines 30-35. 

Blaine Bell, Steven Feiner, and Tobias Hollerer, View Management for Virtual 
and Augmented Reality, 2001, Proceedings of the 14th annual ACM symposium on 
User interface software and technology, pages 101-110, teaches an augmented reality 
system that uses object constraints to control the layout and transparency of objects. 

Daniel G. Aliaga, Virtual Objects in the Real World, 1997, Communications of the 
ACM, pages 49-54, teaches an augmented reality system that determines the locations 
of static real world objects in order to create convincing interactions between the virtual 
objects and the real world objects. 



Application/Control Number: 10/615,941 



Page 19 



Art Unit: 2672 

9. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Jeffery A Brier whose telephone number is 703-305- 
4723. The examiner can normally be reached on M-F from 6:30 to 3:00. If attempts to 
reach the examiner by telephone are unsuccessful, the examiner's supervisor, Michael 
Razavi, can be reached at (703) 305-4713). The fax phone number for the organization 
where this application or proceeding is assigned is 703-872-9306. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). 



Jeffery A Brier 
Primary Examiner 
Art Unit 2672 




